Stability Analysis for Regularized Least Squares Regression

We discuss stability for a class of learning algorithms with respect to noisy labels. The algorithms we consider are for regression, and they involve the minimization of regularized risk functionals, such as L(f) := 1 N PN i=1(f(xi) yi)+ kfkH. We shall call the algorithm ‘stable’ if, when yi is a noisy version of f (xi) for some function f 2 H, the output of the algorithm converges to f as the regularization term and noise simultaneously vanish. We consider two flavors of this problem, one where a data set of N points remains fixed, and the other where N ! 1. For the case where N ! 1, we give conditions for convergence to fE (the function which is the expectation of y(x) for each x), as ! 0. For the fixed N case, we describe the limiting ’non-noisy’, ’non-regularized’ function f , and give conditions for convergence. In the process, we develop a set of tools for dealing with functionals such as L(f), which are applicable to many other problems in learning theory. keywords statistical learning theory, learning in the limit, regularized least squares regression, RKHS